-
Write something in the document below!
- There is at least one public document in every node in the Agora. Whatever you write in it will be integrated and made available for the next visitor to read and edit.
- Write to the Agora from social media.
-
Sign up as a full Agora user.
- As a full user you will be able to contribute your personal notes and resources directly to this knowledge commons. Some setup required :)
Natural Language Processing
Go to [[Week 2 - Introduction]] or back to the [[Main AI Page]]
Tagging of sentence parts forms the basis of most NLP functions.
Rules-based doesn't scale to the enormous datasets present today, so statistic-based tagging and tokenisation must be used.
Methods like an [[Hiden Markov Model]] (HMM) are used independently of humans by the computer to work out the context of each word within a sentence to identify the approproate part of speech, tag, or lemma for that word, depending on the task.
These HMMs use bigrams or n-grams of the words around the masked word to determine the probality of what that word might be. There are multiple methods of masking, some more effective than others. These are especially useful in text generation:
>>> sentence = "the man we saw saw a saw"
>>> tokens = nltk.word_tokenize( sentence )
>>> list(nltk.bigrams( tokens ) )
[('the', 'man'), ('man', 'we'), ('we', 'saw'), ('saw', 'saw'),
('saw', 'a'), ('a', 'saw')]
>>>
Extension Pages
natural language processing
Go back to the [[AI Glossary]]
Natural Language Processing is the domain within machine learning dealing with the tasks required for a machine to work with natural language at a level of abstraction close to that, or at, which humans use it.
These tasks include:
- tokenisation
- tagging
- lemmatisation
- dependancy identification and parsing trees
- shape identification
- part-of-speech recognition
- named entity recognition (NER)
Technologies famous within NLP include:
#ToDo Find a way to use [[Natural Language Generation]] to generate my own [[articles]], most likely about [[Economics]].
Natural Language Processing
- A study of software that aims to parse organic, spoken languages.